Audio-Visual Self-Supervised Terrain Type Recognition for Ground Mobile Platforms

The ability to recognize and identify terrain characteristics is an essential function required for many autonomous ground robots such as social robots, assistive vehicles, exploration robots. Recognizing identifying challenging because similar terrains may have very different appearances (e.g., carpet comes in colors), while with appearance physical properties mulch versus dirt). In order address the inherent ambiguity vision-based recognition identification, we propose a multi-modal self-supervised learning technique that switches between audio features extracted from microphone attached underside of mobile platform image by camera on cluster types. labels are then used train image-based real-time CNN (Convolutional Neural Network) predict types changes. Through experiments, demonstrate proposed type method achieves over 80% accuracy, which greatly outperforms several baselines suggests strong potential applications.